Environmental aware witness for active-active storage cluster nodes

ABSTRACT

The system, devices, and methods disclosed herein relate to a dynamic, robust method for choosing a “winner” in an active-active data storage network. In the systems and methods disclosed herein, two or more intelligent nodes within an active-active data storage network periodically exchange operational parameters in an ongoing negotiation regarding who should be the winner in the event of a communication failure within the network. The winner is chosen dynamically based on the operational parameters. A witness is kept apprised of the winner. In the event of a communication failure between the two nodes, the winner is chosen by the witness based on the most recently negotiated lock file reported by one or both of the nodes.

FIELD OF THE INVENTION

This disclosure is related to the field of data storage and, moreparticularly, to systems and methods for improving witness designationsfor active-active data storage systems.

BACKGROUND

In current storage networks, and particularly storage networks includinggeographically remote directors (or nodes) and storage resources,preserving or reducing bandwidth between resources and directors whileproviding optimized data availability and access is highly desirable.Data access may be localized, in part, to improve access speed to pagesrequested by host devices. Caching pages at directors provideslocalization, however, it is desirable that the cached data be keptcoherent with respect to modifications at other directors that may becaching the same data. An example of a system for providing distributedcache coherence is described in U.S. Patent App. Pub. No. 2006/0031450to Unrau et al., entitled “Systems and Methods for Providing DistributedCache Coherency,” which is incorporated herein by reference. Othersystems and techniques for managing and sharing storage array functionsamong multiple storage groups in a storage network are described, forexample, in U.S. Pat. No. 7,266,706 to Brown et al. entitled “Methodsand Systems for Implementing Shared Disk Array Management Functions,”which is incorporated herein by reference.

Data transfer among storage devices, including transfers for datareplication or mirroring functions, may involve various datasynchronization processing and techniques to provide reliable protectioncopies of data among a source site and a destination site. Insynchronous transfers, data may be transmitted to a remote site and anacknowledgement of a successful write is transmitted synchronously withthe completion thereof.

In an active/active storage system, if there are multiple interfaces toa storage device, each of the interfaces may provide equal access to thestorage device. With active/active storage access, hosts in differentlocations may have simultaneous read/write access via respectiveinterfaces to the same storage device. Various failures in anactive/active system may adversely impact synchronization and hinder theability of the system to recover. Especially problematic are failurescenarios in active/active storage systems involving asynchronous datatransmissions.

Specifically, in active-active data storage environments, it isnecessary to designate a witness to resolve split-brain situations. Asplit brain situation can occur when communication between the variousstorage nodes is lost. In this type of situation, the witness acts as amediator by choosing one of the storage nodes as a winner and making theother a loser. The winning storage node continues to be available, whilethe losing storage node suspends its availability for I/O requests.

At the moment of failure, it is important to choose the best storagenode as the winner because storage nodes may have differentconfigurations and state characteristics at the moment of failure. Intoday's technology, witness selection relies on periodic state exchangemessages as the sole characteristic in choosing which node should takeover in the event of communication loss between active-active nodes.

Witness technology available today or implemented by storage arrayvendors does not take into account the overall availability criteria ofone node versus another node when determining who should be the winner.Current implementations of witness technology only focuses on the healthof the local active-active arrays and their ability to communicate withthe witness itself and the remote node in the event of system or networkfailure.

Witness technology fails to account for the “overall characteristics” ofone node when compared with another node. For example, one node may havea valid data replication leg, more CPU horsepower, more memory banks,and the like. There is thus a need for witness technology to make morerobust decisions when choosing a winning node to be used in failovermode.

SUMMARY

The following Summary and the Abstract set forth at the end of thisapplication are provided herein to introduce some concepts discussed inthe Detailed Description below. The Summary and Abstract sections arenot comprehensive and are not intended to delineate the scope ofprotectable subject matter that is set forth by the claims presentedbelow. All examples and features mentioned below can be combined in anytechnically possible way.

The systems and methods disclosed herein are used when communication hasbeen lost between active-active nodes. When this occurs, the witness isasked to grant the winner role to one of the nodes. The grant assignmenthappens based on the formerly negotiated preferred winner. A preferredwinner negotiation exchange is constantly transpiring between the nodesbased on each node's operational parameters so long as the nodes cancommunicate with each other. The result of the latest negotiation isused during the node to node communication failure to arbitrate thewinner with the witness.

BRIEF DESCRIPTION OF THE DRAWINGS

Objects, features, and advantages of embodiments disclosed herein may bebetter understood by referring to the following description inconjunction with the accompanying drawings. The drawings are not meantto limit the scope of the claims included herewith. For clarity, notevery element may be labeled in every figure. The drawings are notnecessarily to scale, emphasis instead being placed upon illustratingembodiments, principles, and concepts. Thus, features and advantages ofthe present disclosure will become more apparent from the followingdetailed description of exemplary embodiments thereof taken inconjunction with the accompanying drawings in which:

FIG. 1 is an architectural diagram of a network configuration that maybe used in accordance with embodiments of the system described herein.

FIG. 2 is a schematic illustration showing a distributed storage systemaccording to embodiments of the system described herein.

FIG. 3 is a schematic illustration of a portion of a distributed storagesystem according to embodiments of the system described herein.

FIG. 4 is a flow diagram of method steps according to embodimentsdescribed herein.

DETAILED DESCRIPTION

FIG. 1 shows a network configuration 50 that may be used in accordancewith an embodiment of the system described herein. As shown, a pluralityof host devices 10 (10 1 to 10 N) are communicably coupled with aplurality of access nodes 20 (20 1, 20 2 to 20 N). Each of the accessnodes 20 may include a processor (CPU) component 22, such as amicroprocessor or other intelligence module, a cache component 24 (e.g.,RAM cache), an instance of a distributed cache manager 26 and/or otherlocal storage and communication ports. (In general, “N” is used hereinto indicate an indefinite plurality, so that the number “N” whenreferred to one component does not necessarily equal the number “N” of adifferent component. For example, the number of hosts 10 does not, butmay, equal the number of access nodes 20 in FIG. 1.) Cache memory may beconsidered memory that is faster and more easily accessible by aprocessor than other non-cache memory used by a device.

Each of the hosts 10 may be communicably coupled to one or more ofaccess nodes 20 over one or more network connections 15. It is notedthat host devices 10 may be operatively coupled with access nodes 20over any of a number of connection schemes as required for the specificapplication and geographical location relative to each of the accessnodes 20, including, for example, a direct wired or wireless connection,an Internet connection, a local area network (LAN) type connection, awide area network (WAN) type connection, a VLAN, a proprietary networkconnection, a Fibre channel (FC) network etc.

Each of the access nodes 20 may also include, or be communicably coupledwith, one or more volume management functions (VMFs), and may becommunicably coupled with one or multiple storage resources 40, 41, eachincluding one or more disk drives and/or other storage volume, over oneor more storage area networks (SAN) 30, and/or other appropriatenetwork, such as a LAN, WAN, etc. The access nodes 20 may be located inclose physical proximity to each other or one or more may be remotelylocated, e.g., geographically remote, from other access nodes. Each ofthe access nodes 20 may also be able to, intercommunicate with otheraccess nodes over a network 25, such as a private network, an IPnetwork, and/or a Fibre channel (FC) network.

In other embodiments, the access nodes may also be able to communicateover the SAN 30 and/or over the network 15. Several of the access nodes20 may be grouped together at one or more sites in connection with themultiple storage resources 40, 41 and in which the sites aregeographically distant from one another. The system described herein maybe used in connection with a VPLEX product produced by EMC Corporationof Hopkinton, Mass. The system described herein may also be used inconnection with a Remote Data Facility (RDF) storage product produced byDell EMC Corporation, such as a Symmetrix product, a VMAX product, andthe like. Although discussed and illustrated in connection withembodiment for a distributed storage system, the system described hereinmay generally be used in connection with any appropriate distributedprocessing system.

Each distributed cache manager 26 may be responsible for providingcoherence mechanisms for shared data across a distributed set of accessnodes. In general, the distributed cache manager 26 may include a modulewith software executing on a processor or other intelligence module(e.g., ASIC) in an access node. The distributed cache manager 26 may beimplemented in a single access node or distributed across multipleintercommunicating access nodes.

In certain aspects, each of the access nodes 20 may be embodied as acontroller device, or blade, communicably coupled to the storage network30 that allows access to data stored on the storage network. However, itmay be appreciated that a access node may also be embodied as anintelligent fabric switch, a hub adapter and/or other appropriatenetwork device. Because Locality Conscious Directory Migration (LCDM) isapplicable to databases, any suitable networked compute node may beconfigured to operate as an access node with distributed cache managerfunctionality. For example, a distributed cache manager may be run on adesktop computer with a network connection.

According to the system described herein, a distributed storage systemmay enable a storage device to be exported from multiple distributedaccess nodes, which may be either appliances or arrays, for example.With an active/active storage system, hosts in different locations mayhave simultaneous write access to same exported storage device through alocal front-end thereof (i.e., an access node). The distributed storagesystem may be responsible for providing globally consistent and coherentdata access. A witness facility/node, as further discussed elsewhereherein, may enable the distributed storage system to meet consistencyguarantees and maximize data access even when individual front-endsand/or their interconnects fail.

FIG. 2 is a schematic illustration showing a distributed storage system100 according to an embodiment of the system described herein. Thesystem may include a plurality of multiple compute sites, such as site A120 and site B 121, that may each include one or more access nodes likethe access nodes 20 discussed elsewhere herein. The plurality ofmultiple compute sites form a collection of sites. Although two computesites are shown, more than two compute sites may be used and operated inconnection with the system described herein. As further discussedelsewhere herein, the sites 120, 121 may be located geographicallydistant from each other. In an embodiment, the distributed storagesystem 100 may operate in an active/active mode in which multiple sitesmay provide mirroring for at least some of the data and may handleprocessing of host read/write I/O requests.

The sites 120, 121 may be connected via an interconnection 125, such asan IP network, an FC network and/or any other appropriate type ofnetwork. A host 110 is shown that may communicate with the compute sites120, 121 via a network 115. The host 110 is representative of any numberof hosts that may access the distributed storage system 100 via thenetwork 115.

The multiple compute sites 120, 121 may together be engaged in a jointcomputation in connection with I/O handling requests of one or morehosts like the host 110 and maintaining a consistent global data cachetherebetween. The joint computation may include facilitating datamirroring (including possibly allowing write access to multiple sites)as well as managing access to the mirrored data. In another embodiment,the joint computation may include a distributed cache coherence protocolwhich guarantees data consistency for a distributed active/activestorage system. The sites 120, 121 may be coupled via SANs 130, 131 tostorage resources 140, 141.

The storage resources 140, 141 may be located in proximity to the sites120, 121 and/or may be remotely located and accessed. In an embodiment,the SANs 130, 131 may be separate networks. Alternatively, in anotherembodiment, the SANs 130, 131 may be part of the same network, anembodiment shown represented by a dashed line connecting the SANs 130,131. In various embodiments, the joint computation may include multiple,independent sub-computations and may include operations of a clusteredsmall computer system interface (SCSI) device corresponding to use ofexternal storage nodes that may be accessed by one or more of the sites120, 121.

The witness node 160 may include a processor (CPU) and other computingcomponents (e.g., a virtual machine) or connection components suitableto perform the functions and features described herein. The witness node160 may be connected to the sites 120, 121 via a network 165. To improveeffectiveness and availability, the witness node 160 may be deployed inan independent fault domain. In addition, there may be more than onewitness node 160 in alternate embodiments.

A fault domain is a collection of entities impacted by a given fault.For example, to protect against a given fault of a site disaster, thewitness node may be placed at site that is geographically remote fromthe sites 120, 121. Otherwise, a site disaster event may affect the sitewith the witness, rendering it unavailable and leading to suspension(freezing) of the joint computation at the surviving site. Further, forexample, to respond to intersite communication failures, the network 165that connects the witness node 160 to the sites 120, 121 may bedifferent than the intersite connection link 125. The given faults to beprotected by use of the witness node may depend on customer needs andpreferences. In an embodiment, the witness node may be deployed on oneor more hosts like that host 110.

One of the responsibilities of the witness 160 is to prevent asplit-brain mode in which multiple sites act individually with respectto handling host I/O requests (e.g., write requests) in a manner whichmay cause data divergence between sites and/or violate globalconsistency of data and/or individually perform computations which maycause the joint computation to become disjointed and the resultsirreconcilable.

FIG. 3 is a schematic illustration of a portion 300 of the distributedstorage system 100, which will be used to describe embodiments herein.It should be understood that embodiments herein may contain architecturesimilar to that depicted in FIGS. 1 and 2.

FIG. 3 shows portion of an active-active distributed data processingnetwork 300 having a first intelligent storage node 310 and a secondintelligent storage node 320. The first 310 and second intelligentstorage nodes 320 are communicatively coupled via communication link325. We use the term “intelligent” in the context of the storage nodesto indicate that the nodes have processing and memory capabilitiesassociated therewith, either integrated into the hardware of the nodesthemselves, externally located, but communicatively coupled to thenodes. The first 310 and second intelligent storage nodes 320 containprocessing capabilities and memory (not shown) sufficient enough to beable to exchange and store information between themselves.

In addition, the active-active network 300 further comprises a witness360, which could be a stand-alone node or processing unit in someembodiments. In alternate embodiments, witness 360 could be locatedwithin the processing capabilities of first intelligent storage node 310or second intelligent storage node 320. In yet alternate embodiments,there could be multiple witnesses 360. In yet additional embodiments,the one or more witnesses 360 could be located in different networks inorder to enhance fault tolerance.

Irrespective of the exact physical location of witness, witness would becommunicatively coupled to first intelligent storage node 310 viacommunication link 326 and second intelligent storage node 320 viacommunication link 327. In addition, illustratively, the active-activenetwork of FIG. 3 also includes a data replication node 330, also calleda third storage node herein, which is communicatively coupled to thesecond intelligent storage node 320 via communication link 335.

Communication links 325 and 335 can be a synchronous connection.Communication links 326 and 327 may be also be synchronous orasynchronous communication links.

Embodiments herein are directed to overcoming the situation that occurswhen the communication link 325 between the first intelligent storagenode 310 and the second intelligent storage node 320 is degraded below athreshold value. This situation is to be avoided in an active-activedata storage network because first intelligent storage node 310 andsecond intelligent storage node 320 should, ideally, be mirror images ofone another. If their ability to copy data from one to the other iscompromised, any hosts seeking access to the active-active network 300would need to be informed that they should alter their read/write pathinto the network 300.

FIG. 4 depicts method steps associated with embodiments for dynamicallyassigning a winning node in the event of degraded communication betweenfirst 310 and second intelligent storage nodes 320. FIG. 4 furtherillustrates steps performed in system and device embodiments as well.

As can be seen in FIG. 4, the first intelligent storage node 310 and thesecond intelligent storage node 320 periodically exchange operationalparameters 410 with one another. This exchange facilitates a negotiationbetween the two nodes 310, 320 as to which node should be designated as“winner” in the event the quality of the communication link 325 fallsbelow a threshold value. In order to determine which node 310, 320 wouldbe the winner, at least two operational parameters of the nodes 310, 320are taken into consideration.

The winner designation is saved in a lock file, which is sent 414 towitness 360. In some embodiments, the lock file can also containinformation regarding operational parameters, which could also be sent414 to the witness 360. If the communication between the nodes is lost416, the witness arbitrates 418 the winner based on the last receivedlock file. If the quality measurement of the communication link does notfall below a threshold, the nodes 310, 320 continue to exchangeoperational parameters in order to negotiate which should be the winner.

Each node 310, 320 will have a specific lock containing its operationalparameters. Operational parameters can be configuration settings, statecharacteristics, and the like. In some embodiments, one of the nodes310, 320 will have been designated as biased toward being the winner.Assuming in one embodiment the first intelligent storage node 310 wasbiased to be the winner, it could send its lock to witness 360 beforesecond intelligent storage node 320 sends its lock. In this way, witness360 will be made aware of which node 310, 320 was biased toward beingwinner in the event of a communication failure. The nodes 310, 320 or inalternate embodiments, witness 360, may consider information related tobias as one of the criteria it uses in deciding which node 310, 320 toappoint as winner. Additional operational parameters could be, withoutlimitation, a synchronous communication connection to a third storagenode, an asynchronous communication connection to a third storage node,a bias role, a non-bias role, a data replication pathway, a memory boardstate, or a faulted hardware state.

In some embodiments, active-active networks 300 could be customized toweight different operational parameters differently, thereby givingpreference to some operational parameters over others.

In some instances, witness 360 will receive a lock from each active node310, 320. If, however, one of the nodes was impaired significantlyenough to prohibit its ability to send a lock to witness 360, witness360 may have to determine the winner without having the benefit ofreceiving all locks from all nodes 310, 320. Although FIG. 3 shows twonodes 310, 320, in alternate embodiments, there could be more than twonodes 310, 320.

Throughout the entirety of the present disclosure, use of the articles“a” or “an” to modify a noun may be understood to be used forconvenience and to include one, or more than one of the modified noun,unless otherwise specifically stated.

Elements, components, modules, and/or parts thereof that are describedand/or otherwise portrayed through the figures to communicate with, beassociated with, and/or be based on, something else, may be understoodto so communicate, be associated with, and or be based on in a directand/or indirect manner, unless otherwise stipulated herein.

Various changes and modifications of the embodiments shown in thedrawings and described in the specification may be made within thespirit and scope of the present invention. Accordingly, it is intendedthat all matter contained in the above description and shown in theaccompanying drawings be interpreted in an illustrative and not in alimiting sense. The invention is limited only as defined in thefollowing claims and the equivalents thereto.

What is claimed is:
 1. A method for dynamically assigning a winning nodein an active-active data storage network comprising the steps of: a.periodically exchanging operational parameters between two nodes in anactive-active data storage network; b. negotiating between the two nodeswhich would be the winner in the event a quality measurement of acommunication link between the two nodes falls below a threshold value,wherein the negotiation includes evaluating at least two operationalparameters of each individual node; c. choosing a winner based on the atleast two operational parameters of each individual node; d. recording awinner in a lock file; and e. transmitting the lock file to a witness.2. The method of claim 1 wherein the two or more operational parametersare accorded differing priorities.
 3. The method of claim 1 wherein thetwo or more operational parameters are one or more of a synchronouscommunication connection to a third storage node, an asynchronouscommunication connection to a third storage node, a bias role, anon-bias role, a data replication pathway, a memory board state, or afaulted hardware state.
 4. The method of claim 1 wherein the qualitymeasurement is a split-brain state.
 5. The method of claim 1 furthercomprising communicating the winner to at least one of the nodes afterthe quality threshold of the communication link has fallen below athreshold.
 6. The method of claim 1 wherein the lock file furthercontains at least two operational parameters for at least one node.
 7. Asystem for managing an active-active distributed data processingnetwork, comprising: a. A first intelligent storage node and a secondintelligent storage node in an active-active network configurationconnected via a communication link, wherein the first intelligentstorage node and the second intelligent storage node are configured tomonitor a quality measurement of the communication link; b. a witnessnode communicatively coupled to the first intelligent storage node andthe second intelligent storage node; and c. a processor having logicstored thereon configured to: i. periodically exchange operationalparameters between two nodes in an active-active data storage network;ii. negotiate between the two nodes which would be the winner in theevent a quality measurement of a communication link between the twonodes falls below a threshold value, wherein the negotiation includesevaluating at least two operational parameters of each individual node;iii. choose a winner based on the at least two operational parameters ofeach individual node; iv. record at least the two operational parametersfor each individual node and a winner in a lock file; and v. transmitthe lock file to a witness.
 8. The system of claim 7 wherein the two ormore operational parameters are accorded differing priorities.
 9. Thesystem of claim 7 wherein the two or more operational parameters are oneor more of a synchronous communication connection to a third storagenode, an asynchronous communication connection to a third storage node,a bias role, a non-bias role, a data replication pathway, a memory boardstate, or a faulted hardware state.
 10. The system of claim 7 whereinthe quality measurement is a split-brain state.
 11. The system of claim7 wherein the processor is further configured to communicate the winnerto at least one of the nodes after the quality threshold of thecommunication link has fallen below a threshold.
 12. The system of claim7 wherein the lock file further contains at least two operationalparameters for at least one node.
 13. A non-transitory computer readablestorage medium containing software for dynamically assigning a winningnode in an active-active data storage network comprising performing thesteps of: a. periodically exchanging operational parameters between twonodes in an active-active data storage network; b. negotiating betweenthe two nodes which would be the winner in the event a qualitymeasurement of a communication link between the two nodes falls below athreshold value, wherein the negotiation includes evaluating at leasttwo operational parameters of each individual node; c. choosing a winnerbased on the at least two operational parameters of each individualnode; d. recording at least the two operational parameters for eachindividual node and a winner in a lock file; and e. transmitting thelock file to a witness.
 14. The non-transitory computer readable storagemedium of claim 13 wherein the two or more operational parameters areaccorded differing priorities.
 15. The non-transitory computer readablestorage medium of claim 13 wherein the two or more operationalparameters are one or more of a synchronous communication connection toa third storage node, an asynchronous communication connection to athird storage node, a bias role, a non-bias role, a data replicationpathway, a memory board state, or a faulted hardware state.
 16. Thenon-transitory computer readable storage medium of claim 13 wherein thequality measurement is a split-brain state.
 17. The non-transitorycomputer readable storage medium of claim 13 further comprisingcommunicating the winner to at least one of the nodes after the qualitythreshold of the communication link has fallen below a threshold. 18.The non-transitory computer readable storage medium of claim 13 whereinthe lock file further contains at least two operational parameters forat least one node.